Parametric Sequence to Sequence Alignment

This paper presents an approach for establishing correspondences in time and in space between two di erent video sequences of the same dynamic scene recorded by stationary uncalibrated video cameras The method simultaneously estimates both spatial alignment as well as temporal synchronization temporal alignment between the two sequences using all available spatio temporal information Temporal variations between image frames such as moving objects or changes in scene illumination are powerful cues for alignment which cannot be exploited by standard image to image alignment techniques We show that by folding spatial and temporal cues into a single alignment framework situations which are inherently ambiguous for tradi tional image to image alignment methods are often uniquely resolved by sequence to sequence alignment We also present a direct method for sequence to sequence alignment The algorithm simulta neously estimates spatial and temporal alignment parameters directly from measurable sequence quantities without requiring prior estimation of point correspondences frame correspondences or moving object detection Results are shown on real image sequences taken by multiple video cameras A shorter version of this paper will appear in CVPR Introduction The problem of image to image alignment has been extensively studied in the literature By image to image alignment we refer to the problem of densely estimating point correspondences between two or more images either taken by a single moving camera or by multiple cameras i e for each pixel x y in one image nd its corresponding pixel in the other image x y x u y v where u v is the spatial displacement This paper addresses a di erent problem the problem of sequence to sequence alignment which establish correspondences both in time and in space between multiple sequences as opposed to multiple images Namely for each pixel x y in each frame time t in one sequence nd its corresponding frame t and pixel x y in the other sequence x y t x u y v t w where u v w is the spatio temporal displacement The need for sequence to sequence alignment exists in many real world scenarios where multi ple video cameras record information about the same scene over a period of time Some examples are News items commonly documented by several media crews sports events covered by at least a dozen cameras recording the same scene from di erent view points wide area surveillance of the same scene by multiple cameras from di erent observation points Grimson et al sug gested a few applications of multiple collaborating sensors Reid and Zisserman combined information from two independent sequences taken at the th World Cup to resolve the contro versy regarding the famous goal They manually synchronized the sequences and then computed spatial alignment between selected corresponding images i e image to image alignment This is an example where spatio temporal sequence to sequence alignment may provide enhanced alignment Image to image alignment methods e g are inherently restricted to the information contained in individual images the spatial variations within an image which corresponds to scene appearance However a video sequence contains much more information than any individual frame does Scene dynamics such as moving object changes in illumination etc is a property that is inherent to the scene and is thus common to all sequences taken from di erent video cameras It therefore forms an additional powerful cue for alignment Stein and later Lee et al proposed an elegant approach to estimating spatio temporal correspondences between two sequences based on alignment of trajectories of moving objects Centroids of moving objects were detected and tracked in each sequence Spatio temporal alignment parameters were then seeked which would bring the trajectories in the two sequences into alignment No static background information was used in this step This approach is hence referred to in our paper as trajectory to trajectory alignment Giese and Poggio also used trajectory to trajectory alignment to classify human motion patterns Both reported that using temporal information i e the trajectories alone for alignment across the sequences may not su ce and can often lead to inherent ambiguities between temporal and spatial alignment parameters This paper proposes an approach to sequence to sequence alignment which simultaneously uses all available spatial and temporal information within a sequence We show that when there is no temporal information present in the sequence our approach reduces to image to image alignment However when such information exists it takes advantage of it Similarly we show that when no static spatial information is present our approach reduces to trajectory to trajectory alignment Here too when such information is available it takes advantage of it Thus our approach to sequence to sequence alignment combines the bene ts of image to image alignment with the bene ts of trajectory to trajectory alignment and is a generalization of both approaches We show that it resolves many of the inherent ambiguities associated with each of these two classes of methods In a later step re nes the spatial alignment using static background information However the temporal alignment is already xed at that point